fix(deployment): join background deploy thread to stop cross-test state leak - #361
Open
deanq wants to merge 2 commits into
Open
fix(deployment): join background deploy thread to stop cross-test state leak#361deanq wants to merge 2 commits into
deanq wants to merge 2 commits into
Conversation
|
Capy auto-review is paused for this organization because the usage-cycle auto-review limit has been reached. Increase the limit or turn it off in billing settings to resume automatic reviews. |
3 tasks
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes an intermittent CI failure caused by deploy_all_background() spawning a daemon thread without returning a handle to join, allowing mocked state to leak across tests via ResourceManager’s class-level cache.
Changes:
- Update
DeploymentOrchestrator.deploy_all_background()to return a joinablethreading.Thread(orNonefor an empty resource list). - Update/extend unit tests to join the background deploy thread within the patch scope and to assert the new return contract.
- Align
uv.lock’srequires-pythonconstraint withpyproject.tomlby allowing Python 3.13 (<3.14).
Reviewed changes
Copilot reviewed 2 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| uv.lock | Updates the lock metadata requires-python to match supported Python versions. |
| tests/unit/test_deployment.py | Joins the returned background deploy thread in tests and adds coverage for the joinable-thread contract. |
| src/runpod_flash/core/deployment.py | Returns an optional thread from deploy_all_background() so callers (notably tests) can join and avoid cross-test state leaks. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
|
||
| # Should not block | ||
| orchestrator.deploy_all_background(mock_resources) | ||
| thread = orchestrator.deploy_all_background(mock_resources) |
deploy_all_background spawned a daemon thread and returned nothing, so callers had no way to wait for it. test_deploy_all_background patched ResourceManager.get_or_deploy_resource, started the thread, and returned immediately -- the patch unwound while the worker was still running, so the thread went on to hit the real deploy path with AsyncMock resources and cached them via _add_resource. ResourceManager._resources is a class variable, and the late write lands in whichever dict is current when it happens, i.e. one belonging to a later test. The autouse reset_singletons fixture cannot prevent this -- the write occurs after the reset. Any test that subsequently triggers _save_resources() dies in cloudpickle: _pickle.PicklingError: args[0] from __newobj__ args has the wrong class Which test pays for it depends on thread scheduling and on how xdist distributes work, which is why this surfaced as an intermittent failure in test_regressions.py::TestREG008 on a single Python version. Return the thread so callers can join it, and join it in the test inside the patch context. - deploy_all_background now returns Optional[threading.Thread] - test_deploy_all_background joins before releasing its patches - add test_deploy_all_background_returns_joinable_thread to pin the contract, asserting the deploy mock absorbed every resource - assert the empty-list path returns None
pyproject.toml declares >=3.10,<3.14 and CI runs a 3.13 job, but the committed lock still pinned >=3.10,<3.13. Any `uv sync` on 3.13 regenerated the file, leaving a spurious diff in the working tree. Resolution is unchanged -- `uv lock` rewrites only the requires-python line, no package versions move.
deanq
force-pushed
the
deanq/fix-resource-manager-test-pollution
branch
from
August 10, 2026 16:42
cd5f32a to
dc6983a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the intermittent CI failure
which has been hitting a single Python version at random on unrelated PRs.
FAILED <nodeid> - <exc>means the test itself raised, not xdist failing to serialize a report — the victim test is innocent.Root cause
deploy_all_background()spawns a daemon thread and returns nothing, so callers have no handle to join.test_deploy_all_backgroundpatchedget_or_deploy_resource, started the thread, and returned immediately. The patch unwound while the worker was still running, so the thread went on to hit the real deploy path withAsyncMockresources and cached them via_add_resource.ResourceManager._resourcesis a class variable. The late write lands in whichever dict is current when it happens — one belonging to a later test._save_resources()cloudpickles that leftoverAsyncMockand dies.The autouse
reset_singletonsfixture is already correct and function-scoped; it cannot help, because the rogue write happens after the reset. Which test pays depends on thread scheduling and xdist load distribution — hence the randomness.Traced rather than inferred, via a plugin recording which test inserted each
_resourcesentry:Changes
deploy_all_backgroundreturnsOptional[threading.Thread]. Backwards-compatible — it previously returnedNoneimplicitly, and no production caller reads the value.test_deploy_all_backgroundjoins inside the patch context, so the worker can no longer outlive the mock.test_deploy_all_background_returns_joinable_threadpins the contract and asserts the deploy mock absorbed every resource.None.uv.lock'srequires-python(<3.13→<3.14) withpyproject.toml; CI already runs a 3.13 job. Resolution unchanged, no package versions move.Test plan
Written test-first; the new test failed with
AttributeError: 'NoneType' object has no attribute 'join'before the fix, and the pre-fix run visibly logged the leak (caching for cleanup×3), now absent.1 failed, 2629 passed→ now2618 passed, 0 failedmake quality-checkgreen (format, lint, 2618 parallel + 53 serial)mypy src/runpod_flash/core/deployment.pyclean-n autoruns, all2618 passed